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(54) Compensation for video sensor 
movement 

(57) A method and apparatus for 
correcting a video input signal 
received from a scene scanning device 
so that the corrected signal allows for 
the spatial movement of the scanning 
device relative to the scene being 
scanned e.g. the effects of crabbing, 
yawing or pitching of an aircraft 
having a scene scanning device on 
board, are not relayed to the video 
output signal. The corrections are 
applied to the incoming signal before 
storage in a frame store whose output 
may be viewed or recorded. The 
number of lines scanned is greater 
than the number displayed to avoid 
duplication of tines. 
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SPECIFICATION 
Picture correction 



20 



25 



30 



This invention relates to a method and apparatus for processing a video signal to correct for spatial 
inaccuracies or movements of the sensor producing the signal. For example, a vertical- poking cross- 
5 track line scan sensor on boards aircraft scans the ground beneath the a.rcraft along lines ^ra^verse 5 
o the aircraft movement direction. As the aircraft moves, so the scan lines advance along t^ground to 
build up a video signal representative of the ground covered. Thiss.gnal may be d .splayed or recorded .n 
a manner similarto a t.v. video signal. However, if the aircraft is moving in a crabw.se attitude or .f it 
pitches or rolls, or even if it is travelling at an inappropriate height or speed, the m ™*^ 
1 0 signal is eventually displayed and which wiil generally operate on the assumption that the or.g.nal scan 1 0 
° Hnes were evenly spaced along, bisected by and at right angles to the line on the ground above a nc^ I 
along which the aircraft was or is moving, may present a distorted p.cture For example^ he effecl . o 
crabbing or yawing of the aircraft is to produce rhombic distortion of the p.cture wh.le p.tch.ng produces 
successive compression and spreading of the picture. t . . 

1 5 According to the invention, there is provided picture correcting apparatus for receiving a y^deo 1 5 
signal formed by a scene scanning device and for modifying said signal to allow for spatial movement 
and/or repositioning of said device relative to said scene, the apparatus compns.ng frame store means 
. having a data input for receiving successive digitized elements of said s.gna or p.xels ; and h™ng 
address input means for receiving signals in dependence upon which said pixels are stored at respective 
20 addresses within the frame store and the apparatus further compns.ng base address supply means for 
supplying a base address signal for the pixels to be stored in the frame store computer means fo 
calculating, on the basis of positional information relating to said device and/or a craft whe« " «t is 
carried a series of address offset values respectively associated with respective ones of a current group 
o?sa!d pixels, register means for storing said values calculated by said computer, and 1 iiean for reading 
25 said stored values in sequence and for causing a signal comprising a combination of the ea d value and 
- - said base address signal to be passed to said.frame store in synchronism with the ' associated I pixel. 

With the method and apparatus to be particularly described and illustrated here.n, a scanned 
picture can be corrected in real time forskew providing the instantaneous values of error are known or 
may be predicted. The corrected data is stored in a frame store (or partial frame store) and may be 
30 passed on for viewing or storage with a delay of one frame maximum. 

For a better understanding of the invention, reference will now be made, by way of example, to the 

aCC Tgu^ of a picture correction apparatusfor use with an aircraftline 

scanner geometrical diagram for explaining part of the operation of figure 1 . 

The apparatus shown is operable for receiving a video signal V from an aircraft borne verbal- 
looking cross-track line scanner (not shown) and, in dependence upon signals (from the aircraft . 
navigation system for example) indicative of instant values of roll, pitch, crab, yaw spec ,d and height^ of 
the craft, for producing a picture signal corrected as regards these parameters. For the exempli .shown 
40 the line scanner is assumed to comprise say an infra-red radiation detector element or array thereof and 
an optical system including a rotating polygonal reflector element which, as it rotates °™f t ™ p 
radiation received by the detector element(s) to be that which arrives from the ground beneath the 
aircraft in a direction varying between the horizontal to one side of the aircraft and the honzont l a * the 
other side. Thus, in effect the detector elements) scans the ground from one horizon to the other along 
45 lines perpendicular to the direction of movement of the craft The scan lines advance along the ground 
as the aLaft moves giving a continuous frame advancing effect. The scanner .s fixed wlthir .the aircraft 
so that, rf the aircraft rolls, the normal mid-scan position on the ground will ^ove f rom d.rec lyl beneath 
the aircraft to one or the other side thereof. Depending on the aircraft speed and the ^*«'?*™ n 
line on the ground, which in turn depends upon the aircraft height and the convergence of the instant 
™ field of view of the scanner, the lines may overlap one another, be exactly contiguous, or may be 
50 sp ced apTrt 0.e so that some.portions of the ground are missed). If the aircraft pitch vanes then 

while it is'actually changing, some of the scan lines will be compressed ^^^^^X^ 
aircraft 'yaws* or 'crabs', the direction of the lines will vary from the correct position of perpendicular to 
the aircraft movement direction. The effect of the above possibilities, if uncorrected, is to produce 
55 distortion in the eventual displayed picture. 

Referring now to the drawing, the video signal V comprises a series of portions each 
representative of a respective scanning line and each comprising a series of values representative of 
e^^^^urasla^ant. or 'pixels' along the respective line. Also received from the scanner ■» ^ l«e 
sync, signal which is here assumed to be generated at the start of each hne, i.e. whenever the scanner l8 
60 looking in the horizontal direction from which the scan lines start The video and line sync, signals are 
received by a correction logic circuit 1 comprising a plurality of shift registers to be described separately 
later. The output from the logic circuit 1 is passed to a frame-store 2. On the basis of navigation 
information passed thereto from say the aircraft navigation system, a function computer 3 controls the 



40 



45 



55 



60 
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logic circuit 1 and the frame store 2 to select and/or reposition pixels and lines of the video signal V 
whereby a corrected video signal Vo becomes available at the output of the frame store. 

The method of applying the corrections is to calculate the nearest, rectangular coordinate point for 
the data collected and store the data in that particular pixel store in the frame store. 
5 The correction factors are calculated by the function computer and output to the appropriate 5 
registers. Since the correction factors do not change rapidly, they can be loaded and used while the next 
set of correction factors are calculated. 

The corrections are actually performed by the apparatus in three stages as follows: — 



i. Roll distortion. This is simply corrected by adjustment of the line sync, pulse in accordance 
1 0 with the roll angle. This correction is assumed to be performed prior to the "line select" 10 

operation. 

it. Rate/position corrections. The parameters affecting these factors are vehicle height, ground 
speed and pitch. They affect the number of lines (and hence pixels) along the line of track i.e. 
vertically up the display. 

15 iii. Angular corrections. These have two inputs Yaw and Crab. They both have the same effects 15 

and differ only in their time constants, Yaw is relatively rapid while Crab is slow. Crab and 
Yaw angles are added arithmetically to produce the angular corrections in both X and Y 
coordinates. 

Referring now to figure 1 , the video signal V is fed via an And gate 4 to a line select unit 5 of the 
20 logic ie. circuit 1 . As mentioned earlier, the signal V comprises a series of video signal lines each 20 
corresponding to a scan of the line scanner. The function of the line select unit 5 is to pass a controlled 
proportion of the number of received video lines with a view to causing the eventual picture signal to 
contain a desired number of lines for a given distance over the ground, which number of lines are 
reasonably evenly spaced over this distance, despite changes in aircraft ground speed, height and pitch. 
25 Signals G, H and P respectfully representative of the latter parameters are passed to the line select unit 25 
5 from the aircraft navigation apparatus (not shown). The vehicle ground speed will determine how 
many sensor scans are taken over any length of ground. The 'pitch' of the vehicle will effectively alter 
the ground speed (both positively and negatively) with a fairly rapid time constant The pitch angle, 
knowing the height, can be used to determine an instantaneous value of ground speed. 
30 The number of scan lines to be displayed per unit of ground length covered is also known as are 30 

the scan rate of the sensor and the instantaneous ground speed. Therefrom there is determined the ratio 
of the number of lines gathered to the number of lines displayed. The number of lines gathered must 
- always be greater than the number of lines displayed if the duplication of lines (hollow magnification) is 
to be avoided. In practice between 2 and 1 3 lines are gathered for each one displayed. 
35 The number of lines output from the line select operation must be equal to the number of lines 35 
displayed (or an integer multiple of them). 

The lines to be output are selected by finding the nearest line to the correct displayed position, i.e. 
the nearest "line vertical" as follows: — In the following explanation, N is the number of lines scanned 
per Kilometre, n is the number of lines displayed per Kilometre, Iv' is the input line number and Iv is the' 

_40__displayed line number. .„ ... . .40 

The nearest value will change each line unless N is an integer multiple of n, then the offsets will 
cycle at the N/n rate. 

It will not be possible to compute new values at this rate so a compromise will be made. 
If the effect of pitch is not taken into account, the nearest display line to any given input line is 
45 ! given by: 45 



Iv' Vn 2) 



1 (M) 

where a Is the 
vertical difference 
between the two ref. 

50 origins. 50 
so that input lines would only be used or passed on when 



(M) 
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The effect of changing pitch of the aircraft can be taken into account by modifying equation 1 as 
follows: — 

The displacement on the ground = h tan S 

where h is vehicle height and S is the instantaneous pitch angle from the horizontal. 
This gives a pitch factor which is added to equation 1 to give: — 5 

and input lines are now deemed valid and hence used when tv' satisfies the following criteria: — 

■ 1 (RT (Fl T" 

The displayed line number then becomes 

IIV (-&■ + 4) + Q + n{h fan &)/N| 

U .} ..,,.:,,,," 

In effect, the shift caused by the vehicle pitch {$) has been added to the line position. As will be 
appreciated pitch variation introduces an additional variable and it can be catered for easily only 
providing the angle changes relatively slowly. This argument applies also to N which is a function of the 
vehicle ground speed. 

1 5 If the number of lines being output from the 'tine select' operation is an integer multiple of the 1 5 

number being displayed, then a posittonally more accurate picture can be built up at the expense of 
additional hardware. Since an additional Y offset Shift register and X Offset Shift register is required for 
each multiple, e.g. 7 lines 7 registers. 

It may also be possible for fractional lines collected/lines displayed to be handled provided it is 
20 possible to dynamically calculate the X and Y offset values in real time. 20 
The line data collected must be longer than that displayed to allow for the Crab and Yaw 
corrections. 

Line length displayed 

Line Length collected > 

Cosine (Crab angle + Yaw angle} 

If insufficient line length is collected then it will result in blank strips at the edge of the display, for 

25 areas where data was not collected. It will not affect the positional accuracy however. 25 
In order to correct for yaw and crab, the input pixels are displaced vertically, the amount of 
displacement varying with the horizontal position of the pixel, and preferably also horizontally to correct 
for the lateral displacement introduced by the crab and yaw. The latter corrections are only realiy 
relevant for the edges of the picture and are not as important as the vertical corrections. The values of 

30 vertical displacement for the respective pixels of a line are loaded by computer 3 into a Y-offset shift 30 
register 9. The displacement values are read from the register in sequence in synchronism with the 
reading out, from pixel shift register 7, of the pixels themselves. Each displacement value is added to {or 
subtracted from) the line number provided by a line counter 1 0. The content of which is incremented 
by the line sync pulses from the scanner, to give a corrected Y-address value for the relevant pixel. 

35 Meanwhile, in a similar way, horizontal displacement values loaded into an X-offset shift register 1 0 by 35 
computer 3 are being read from that register in sequence and added to (or subtracted from) a value 
supplied by pixel counter 1 2, and incremented by a strain of pixel sync pulses fed to the apparatus, 
which is representative of the base or uncorrected horizontal pixel position. The resultant corrected 
horizontal or X-address value is fed to frame store 2 along with the corrected Y-address value and the 

40 corresponding pixel is stored at that corrected address. The displacement values are calculated by the 40 
computer 3 as follows: — 
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Let scanned lines/km 

displayed lines/km 
input line width 

displayed line width represents 

Vertical Line count from bottom of display 
Horizontal pixel count from left hand edge of display 
Iv & l h are integers. 
Input pixels/km 
displayed pixels/km 

Input pixel width 



= N 



k/N metres 



— = metres 
n 

= Iv 

= l h 

= M 

— m 

k 

— = metres 



10 



displayed pixel width represents 



— = metres 



= e. 



15 



20 



Yaw and Crab angle 

The horizontal input pixel count l h ' is counted from some arbitrary trigger point at the start of the 
scan. The Vertical line count lv' is counted from an arbitrary point in the frame. ^ 

It is assumed that the input data is digitized at a rate that will give consecutive pixels just 

^No^^^ 

are vertically in line with the output pixels. The frequency adjustment is f N = y Sin 9 
where f N = New Sampling frequency 
f Q = display sample frequency 

... 0-=Yaw>"Crab "" ~ - 

Displayed Pixel centres are:— jU /» + Ai 

input Pixel centres are:- ^ ^ £ ^ e * bj (l n U ^)-L S in 6 ♦ ( V ♦ |) ^ *a 



Distance between centres is:- 



Sin 0 



N Cos e 



-J 
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This distance is a minimum when each of the contents of the { } brackets is a minimum, since only real 
numbers are involved. 



Horizontals 



Oh * 1) "ST - Oh' * I) -T 6 " b " MINIMUM 

( J h *t)|" ( r h' * 4") m" Cos 6 * b IF MMMUM ZERO 

I 



DENOTES NEAREST 
INTEGER VALUE 



WHERE 

If pixel quantization is the same vertically and horizontally then M = N m = n 



c os e --!♦-£!- 

2 k. 



Vertically 



(lv l + $)k 
N Cos 8 



q IF MINIMUM ZERO 



Linear magnification and equal resolution vertically and horizontally 



s,n 9 * n N Cos 6 " k 



M=N m=n 

in this system Iv is stepped with iv' if there is only one line selected for one line displayed so the 
1 0 vertical stepping is effectively removed from the correction equation, i.e. I v' = 0 for one line in, one line 1 0 
out. For more than one line in, Iv' takes on the values 0 for the first line, 1 for the second line etc. 



15 



The respective displacement values for all the input pixels from first to last in the line, i.e. the 
respective displacement values for all the values of input pixel position l h ' from 0 to the total number of 

1 5 input pixels in a line, are calculated by the computer using the functions described above and then held 
for as long as the corresponding value of yaw plus crab angle 9 remains extant. If 9 varies by more than 
a predetermined increment, a new set of values are calculated and loaded. 

For expediency, it is preferred that the computer 3 should anticipate each incremental change in 9 
by holding the displacement values associated with the value of Q prior to its last incremental change and 

20 by calculating and holding the displacement values for the value of & which would be extant if it were to 20 
continue to change in the same direction through another increment. Then, if it does so continue to 
change, these newly calculated values can be immediately loaded into the offset registers. On the other 
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hand, if the direction of change of 9 reverses and goes back to the previous incremental value, the held 
displacement values associated with the prior 0 value can be loaded, in this case, of course, each 
recalculation by the computer is not of the displacement values for the present 6 value but of those for 
the next expected value. The extra sets of displacement values may be held in respective further shift 
5 registers (not shown) each comprising a length or number of stages which, like that of the registers 9 5 
and 1 0 equals the maximum numberof pixels per line. 

The width of the Y-offset register7£e- the bit capacity of each stage thereof, and the width of each 
of the further vertical displacement registers if used, should be sufficient to allow for the maximum 
amount of vertical offset likely to be required which in turn depends upon the likely maximum value of 
1 0 yaw plus crab angle 6. For example, if the likely maximum value of B is + 25°, then for a 1 000 x 1 000 1 0 
pixel data display the maximum offset, as a number of lines, is + 1 000 tan 25° or + 466. Ten bits 
would be required for this (giving in fact a maximum offset value range of ± 512). 

The line start unit 6 inhibits Pixel data and pixel sync in dependence upon Roll angle, f.e. r it gets all 
lines to start at the same angle to the air vehicle position rather than its attitude. For example, the data 
1 5 and pixel sync may be inhibited until the angle is such as to give a swathe of 1 km on the ground taking 1 5 
into account the roll angje and the height. 

Figure 2 of the drawings shows the position P of the aircraft looking along its longitudinal or roll 
axis. The aircraft is at height h above the ground G and it is desired that the displayed picture should be 
of an area extending to 500 metres from each side of the line directly beneath the aircraft. As 
20 mentioned earlier, the line sync signal is produced by the line scanner when this is looking in a direction 20 
which would be horizontal if the aircraft were flying straight and level La. when the scanner is looking in 
. a direction parallel to the lateral or pitch axis Y. Meanwhile, the delayed line sync signal should occur 
when the scanner is looking at the first boundary of the area to be displayed, i.e. along the line L if the 
scanning proceeds in the direction shown by arrow A. Thus, the required delay equals the time required 
25 for the scanner rotor to move through the angle X - tan" 1 (h/500) between line L and the horizontal plus 25 
(or minus) the actual angle of roll. If the angular velocity of the scanner rojor is W then the delay is (0 + 
tan" 1 (h/5000). W seconds where O is the roll angle — this is made positive if the roll is opposite the 
direction of scanner rotor movement and negative otherwise. 

Meanwhile, the end of each line occurs after (0 +- tan -1 (h/500) + 2 tan" 1 (500/h)). W seconds 
30 which expression reduces to (0 + nil + tan -1 (500/h)). W seconds from the line sync pulse. 30 
The pixel shift register 7 is designed to smooth out the differing rates of collection and use of the 
pixels. If the hardware can cope with the maximum pixel gathering rate it will not be needed. _ _ ' . 

The device is essentially a F.l.F.O. (First in FirstOut) register. It holds the quantized video data in 
the form of 6 (or 7 or 8) bit pixels. A pixel is output from the register to be stored in the frame store on 
35 every Pixel sync pulse. The position it will be stored in is determined by the current X and Y address, and 35 
whether or not it is to be stored will depend upon the "Not Used" register. 

Since there are more pixels gathered than displayed, no pixel will be used twice. If a pixel had to 
be used twice then another register containing information to use the pixel twice (or more) must be 
used, together with circuitry to inhibit shifting the F.l.F.O. while the X arid Y Offset registers are shifted. 
40 Associated with the register 7 is a 'Pixel not used' register 8 which is used to indicate when any 40 

particular pixel will not be used, this will occur when another pixel is better suited to the display 

-positron, orwhen a. pixel, is discarded, owing.to-wrong positioning, as jn_the.case.of recti linearization 

which may also be performed in this unit. 

The unit is loaded from the computer in conjunction with X and Y offset registers. The pixels are 
45 not used when any two (or more) values of input pixel l' h give the same two values of Iv and l h (this can 45 
happen since Iv and l h are integers). Alt but one of these pixels must be discarded. In fact, the 
occurrence of pixels with the same values of Iv and lh will be quite low and it would not matter too 
much if, on these occasions, a quite arbitrary decision were made about which one is not discarded. 
Preferably, however the one kept is that whose actual values are nearest the integer values. The nearest 
50 and correct choice is given by the value of l^ +n and which gives the iowest value to the function. 50 

{v-s-(wi)cos 9 -i^y 

for values of A = 0, 1, 2 or— 1, —2 etc. The values of l£ which give the same Iv and l h should be 
discarded except the value of l^ which gives the lowest value of the function above. 

As mentioned earlier, the X-offset shift register 1 0 is used to apply any lateral shift to a pixel to 
55 correct for the lateral displacement introduced by the Crab and Yaw, which corrections are only required 55 
at the edges of the picture and are not as important as the Y-offset corrections. 
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The rectilinearization corrections can also be added to this register if desired. The rectilinear 
corrections can be applied by computing a composite value of the Y offsets or holding the rectilinear 
offsets in a separate shift register and adding the two outputs. 

The contents of the register are calculated from: — 



5 



VALUE=Ih'-4l(l h .4).Cos9--l^ 





5 



for each value of l h ' up to the maximum number of pixels per line. 

The width of this register need only be sufficient to cater for the maximum lateral offset of any 
pixel. For a 1000 pixels/line display and maximum yaw and crab angle 8 = 25° (the sign of $ is not 
relevant here — \-2B° and —25° give exactly the same effect): — 

10 Maximum offset (as a number of pixel positions) 10 

= 1000(1 -Cos 25) = 94. 

This will require 8 bits only. 

t 

The frame store 2 accepts data and modified X and Y addresses and stores the pixel value in the 
appropriate position in the same store. This position is now spatially correct. 

1 5 The data may be read out from the frame store as another part is being loaded, providing the store 1 5 
is large enough to cater for the largest values of 6 then the store can be used as a rolling store. Note that 
it does not have to cater for + $ since both values are not likely to occur within one frame. If the frame is 
small then the "offset value for unloading the store must allow for the sign of 0. 

— - -As described.-the computer 3 takes the values of Yaw, Crab,-height,-Groundspeed and pitch, and' - 

20 computes which lines to select and the values of the X and Y offset registers, is also calculates which 20 
pixels are to be used. Since the parameters will not change rapidly in relation to the video data, a fairly 
slow computer should be able to cope with the demand. If, as is likely, the input parameters are 
continuous, they are quantised into steps which have a meaningful size. 

The principle mentioned earlier, whereby the computer holds and/or calculates previous and likely 

25 next yaw and crab displacement values could be extended so that say the previous and likely next set of 25 
values for the line select, X and Y offset registers and the 'Not used' registers are kept available. The 
required set of values can then be loaded into the appropriate register as soon as the relevant parameter 
has undergone an incremental change and, while holding the previous set in case the direction of 
change reverses, the computer can calculate the set appropriate to the next incremental parameter 

30 change in the same direction. 30 
the description given above relates particularly to apparatus for correcting the signal supplied by a 
line-scanner on board an aircraft (which term includes helicopters). Thus, the types of correction applied 
take account of the possible spatial positioning of an aircraft. It will be appreciated however that the 
invention is not only applicable to the described situation. Thus, It could be used for another sort of 

35 scanning sensor, for example a t.v. camera, (particularly forward-looking t.v. surveillance apparatus on 35 
an aircraft) or for a line-scanner or other kind of sensor on board a vehicle or platform other than an 
aircraft. Whether or not the particularly described situation is extant, it may be desirable, necessary or 
allowable to modify the corrections applied to the signal from those described, or possibly not all of 
them will be needed. For example, even for a line-scanner or an aircraft, it may not be necessary to 

40 correct for say roll and recti linearisation since these may be corrected, perhaps preferably so, prior to 40 
the correcting apparatus. 

As will be appreciated, the particular implementation of the hardware making up the apparatus is 
selectable and will be chosen having regard particularly to the input data ratio. and the required size of 
the frame store 2. 

45 CLAIMS 45 
1. Picture correcting apparatus for receiving a video signal formed by a scene scanning device and 
for modifying said signal to allow for spatial movement and/or malpositioning of said device relative to 
said scene, the apparatus comprising frame store means having a data input for receiving successive 
digitized elements of said signal or 'pixels' and having address input means for receiving signals in 

50 dependence upon which said pixels are stored at respective addresses within the frame store and the 50 
apparatus further comprising base address supply means for supplying a base address signal for the 
pixels to be stored in the frame store, computer means for calculating, on the basis of positional 
information relating to said device and/or a craft wherein it is carried, a series of address off-set values 
respectively associated with respective ones of a current group of said pixels, register means for 

55 storing said values calculated by said computer, and means for reading said stored values in sequence 55 
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and for causing a signal comprising a combination of the read value and said base address signal tpjbejQ 19|IbW 
passed to said frame store in synchronism with the associated pixel. VTHOfJA tfiatGS 

2. Picture correcting apparatus substantially as hereinbefore described with reference to the*°« q 
accompanying drawings. J ^ n * ^MIMOn .U" 

3. A picture correcting method substantially as hereinbefore described with reference to tne1 .giudZlSfttjiiS 
accompanying drawings. I 
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